Augmented reality device and operating method thereof

ABSTRACT

Provided are an augmented reality device for improving work concentration and immersion of a user, and an operating method of the augmented reality device. The augmented reality device according to an embodiment of the disclosure may obtain a plurality of image frames by photographing a real scene by using a camera, may recognize a preset object from the plurality of image frames, may obtain a blur image in which an area other than an area of the at least one preset object from among entire area of the plurality of image frames is blurred, and may display the blur image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of InternationalApplication PCT/KR2023/001798 filed on Feb. 8, 2023, which claimsbenefit of Korean Patent Application No. 10-2022-0032943, filed on Mar.16, 2022, at the Korean Intellectual Property Office, the disclosures ofwhich are incorporated herein in their entireties by reference.

TECHNICAL FIELD

The disclosure relates to an augmented reality device for modifying anddisplaying an image related to a real scene within a field of view (FOV)and an operating method of the augmented reality device. Moreparticularly, the disclosure relates to an augmented reality device forblurring and displaying a partial area in a work space of a user and anoperating method of the augmented reality device.

BACKGROUND ART

Augmented reality is a technology to overlay or superimpose a virtualimage on a physical environment space of the real world or a real worldobject, and to display the virtual image together with the real worldobject. An augmented reality device (e.g., smart glasses) usingaugmented reality technology is efficiently used in everyday life, forexample, for information search, direction guidance, and cameraphotographing. Especially, smart glasses, as an example of an augmentedreality device, are worn as a fashion item and mainly used for outdooractivities.

An augmented reality device generally enables a user to view a scenethrough a see-through display arranged close to his or her eyes whilethe user is wearing the augmented reality device. In this case, thescene includes at least one real world object in a physical environmentor a space that the user views directly with his or her eyes. The usermay simultaneously view the real world object and a virtual imagethrough the see-through display of the augmented reality device.

A user may perform a work by using various objects in a work space. Whenthere are multiple objects within a field of view of the user in thework space, objects not directly related to the work from among themultiple objects, for example, a mobile phone or a tablet PC, mayattract the user's attention through a message notification, Internetsearch, Internet shopping, etc. The work of the user may be disturbeddue to the objects not directly related to the work within the field ofview of the user, and user's concentration on the work may be reduced.In order to prevent a decrease in the user's concentration or the user'sdistraction due to objects not related to his/her work, a method ofperforming the work while wearing an augmented reality device may beconsidered.

SUMMARY

The disclosure provides an augmented reality device for improving workconcentration and immersion of a user by blurring an area where at leastone object not related to the work is located, from among a plurality ofobjects within a field of view of the user, and an operating method ofthe augmented reality device. An augmented reality device according toan embodiment of the disclosure may perform image rendering for blurringa peripheral area other than at least one object related to a work, byperforming object detection, object tracking, and object segmentation byusing an artificial intelligence (AI) model.

An embodiment of the disclosure provides an augmented reality deviceincluding: a camera configured to obtain a plurality of image frames ofa scene; a display; a memory storing information about an object; and atleast one processor, wherein the at least one processor is configuredto: recognize at least one object corresponding to the information aboutthe object from a first image frame from among the plurality of imageframes obtained through the camera, obtain a mask by segmenting an areaof the recognized at least one object from the first image frame, obtaina blur image in which an area other than the area of the at least oneobject, is blurred, the blur image being obtained by rendering using themask, a second image frame obtained after the first image frame, andcontrol the display to display the blur image.

In yet another embodiment, there is an operating method of an augmentedreality device, the operating method including: obtaining a plurality ofimage frames by photographing a scene by using a camera; recognizing atleast one object corresponding to information about a object from afirst image frame from among the plurality of image frames; obtaining amask, by segmenting an area of the recognized at least one object fromthe first image frame; obtaining a blur image in which an area otherthan the area of the at least one object is blurred, the blur imagebeing obtained by rendering using the mask, a second image frameobtained after the first image frame; and displaying the blur image.

An embodiment of the disclosure provides an augmented reality deviceincluding a camera configured to obtain a plurality of image frames of areal scene, a display, a memory storing information about a presetobject, and at least one processor, wherein the at least one processoris configured to recognize at least one preset object corresponding tothe information about the preset object from a first image frame fromamong the plurality of image frames obtained through the camera, obtaina mask, by segmenting an area of the recognized at least one presetobject from the first image frame, obtain a blur image in which an areaother than the area of the at least one preset object is blurred, byrendering, by using the mask, a second image frame obtained after thefirst image frame, and control the display to display the blur image.

The at least one processor may be further configured to detect aplurality of objects from the first image frame by using an objectdetection model, and identify the at least one preset object from amongthe plurality of objects, wherein the at least one preset object is auser-defined object pre-defined as an object related to a work of auser.

The at least one processor may be further configured to recognize the atleast one preset object from the first image frame, by tracking at leastone object recognized in an image frame prior to the first image frameby using an object tracking algorithm.

The augmented reality device may further include a user input interfaceconfigured to receive a user input that selects an important object fromamong the recognized at least one preset object, wherein the at leastone processor is further configured to obtain the mask by segmenting,from the first image frame, an area corresponding to the importantobject selected based on the user input received through the user inputinterface.

The at least one processor may be further configured to blur aperipheral area other than an area corresponding to the at least onepreset object from among entire area of the second image frame, byperforming convolution of the second image frame and the mask.

The at least one processor may be further configured to obtain the blurimage, by performing image rendering only during a preset working time,and display the blur image on the display only during the preset workingtime.

The at least one processor may be further configured to synthesize theblurred area with a different color according to a work type or a workenvironment.

The at least one processor may be further configured to synthesize theblurred area with a virtual object or a graphical user interface (GUI)that provides information related to a work.

The augmented reality device may further include a user input interfaceconfigured to receive a user input that determines a blur optionincluding at least one of a blur degree, a color, or a brightness of theblurred area, wherein the at least one processor is further configuredto perform image rendering for blurring the area based on the bluroption determined by the user input received through the user inputinterface.

The augmented reality device may further include a user input interfaceconfigured to receive a user input that selects at least one object fromamong a plurality of objects recognized from the first image frame,wherein the at least one processor is further configured to performimage rendering for blurring the at least one object selected by theuser input received through the user input interface.

Another embodiment of the disclosure provides an operating method of anaugmented reality device. The operating method includes obtaining aplurality of image frames by photographing a real scene by using acamera, recognizing at least one preset object corresponding toinformation about a preset object from a first image frame from amongthe plurality of image frames, obtaining a mask, by segmenting an areaof the recognized at least one preset object from the first image frame,obtaining a blur image in which an area other than the area of the atleast one preset object is blurred, by rendering, by using the mask, asecond image frame obtained after the first image frame, and displayingthe blur image.

The recognizing of the at least one preset object may include detectinga plurality of objects from the first image frame by using an objectdetection model, and identifying the at least one preset object fromamong the plurality of objects, wherein the at least one preset objectis a user-defined object pre-defined as an object related to a work of auser.

The obtaining of the mask may include selecting an important object fromamong the recognized at least one preset object, based on a user input,and obtaining the mask, by segmenting an area corresponding to theselected important area from the first image frame.

The obtaining of the blur image may include blurring a peripheral areaother than an area corresponding to the at least one preset object fromamong entire area of the second image frame, by performing convolutionof the second image frame and the mask.

The obtaining of the blur image may include obtaining the blur image, byperforming image rendering only during a preset working time, and thedisplaying of the blur image may include displaying the blur image onlyduring the preset working time.

The obtaining of the blur image may include synthesizing the blurredarea with a different color according to a work type or a workenvironment.

The obtaining of the blur image may include synthesizing the blurredarea with a virtual object or a graphical user interface (GUI) thatprovides information related to a work.

The obtaining of the blur image may include determining a blur optionincluding at least one of a blur degree, a color, or a brightness basedon a user input, and performing image rendering for blurring the areabased on the blur operation determined by the user input.

The operating method may further include selecting at least one objectfrom among a plurality of objects recognized from the first image framebased on a user input, wherein the obtaining of the blur image includesperforming image rendering for blurring the at least one object selectedby the user input.

Another embodiment of the disclosure provides a computer-readablerecording medium having recorded thereon a program for executing theoperating method, on a computer.

BRIEF DESCRIPTION OF DRAWINGS

The disclosure may be readily understood from the following detaileddescription in conjunction with the accompanying drawings, and referencenumerals denote structural elements.

FIG. 1 is a conceptual view illustrating an operation by which anaugmented reality device displays a blur image of a real scene,according to an embodiment of the disclosure.

FIG. 2 is a block diagram illustrating elements of an augmented realitydevice, according to an embodiment of the disclosure.

FIG. 3 is a flowchart illustrating an operating method of an augmentedreality device, according to an embodiment of the disclosure.

FIG. 4 is a view illustrating an operation by which an augmented realitydevice obtains a blur image, according to an embodiment of thedisclosure.

FIG. 5 is a flowchart illustrating a method by which an augmentedreality device identifies a user-defined object from a first imageframe, according to an embodiment of the disclosure.

FIG. 6 is a view illustrating an operation by which an augmented realitydevice recognizes a user-defined object from a first image frame,according to an embodiment of the disclosure.

FIG. 7 is a view illustrating an operation by which an augmented realitydevice recognizes a user-defined object from a first image frame,according to an embodiment of the disclosure.

FIG. 8 is a view illustrating an operation by which an augmented realitydevice obtains a blur image, according to an embodiment of thedisclosure.

FIG. 9 is a view for describing an operation by which an augmentedreality device displays different images as time passes, according to anembodiment of the disclosure.

FIG. 10 is a view for describing an operation by which an augmentedreality device displays different blur images according to a work type,according to an embodiment of the disclosure.

FIG. 11 is a view illustrating an operation by which an augmentedreality device displays a virtual object or a graphical user interface(GUI) in a blurred area, according to an embodiment of the disclosure.

FIG. 12 is a flowchart illustrating a method by which an augmentedreality device obtains a blur image according to a blur optiondetermined based on a user input, according to an embodiment of thedisclosure.

FIG. 13 is a flowchart illustrating a method of obtaining a blur imageby blurring an object selected by a user input, according to anembodiment of the disclosure.

FIG. 14 is a view illustrating an operation of obtaining a blur image byblurring an object selected by a user input, according to an embodimentof the disclosure.

DETAILED DESCRIPTION

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

The terms used herein are those general terms currently widely used inthe art in consideration of functions in the disclosure but the termsmay vary according to the intention of one of ordinary skill in the art,precedents, or new technology in the art. Also, some of the terms usedherein may be arbitrarily chosen by the present applicant, and in thiscase, these terms are defined in detail below. Accordingly, the specificterms used herein should be defined based on the unique meanings thereofand the whole context of the disclosure.

An expression used in the singular may encompass the expression in theplural, unless it has a clearly different meaning in the context. Termsused herein, including technical or scientific terms, may have the samemeaning as commonly understood by one of ordinary skill in the artdescribed in the disclosure.

When a part “includes” or “comprises” an element, unless there is aparticular description contrary thereto, the part may further includeother elements, not excluding the other elements. The term used in theembodiments such as “unit” or “module” indicates a unit for processingat least one function or operation, and may be implemented in hardware,software, or in a combination of hardware and software.

The expression “configured (or set) to” used in the specification may bereplaced with, for example, “suitable for,” “having the capacity to,”“designed to,” “adapted to,” “made to,” or “capable of” according to asituation. The term “configured to (or set)” does not always mean only“specifically designed to” by hardware. Alternatively, in somesituation, the expression “system configured to” may mean that thesystem is “capable of” operating together with another apparatus orcomponent. For example, “a processor configured (or set) to perform A,B, and C” may be a dedicated processor (e.g., an embedded processor) forperforming a corresponding operation or a generic-purpose processor(such as a central processing unit (CPU) or an application processor)that may perform a corresponding operation by executing at least onesoftware program stored in a memory.

When a component is referred to as being “connected” or “accessed” to orby any other component, it should be understood that the component maybe directly connected or accessed to or by the other component, butanother new component may also be interposed between them, unlessotherwise specifically indicated.

In the disclosure, ‘augmented reality’ means showing a virtual image ina physical environment space of the real world or showing a real worldobject and a virtual image together.

Also, an ‘augmented reality device’ may be a device capable ofrepresenting ‘augmented reality’, and may include, as well as augmentedreality glasses being in the form of glasses that a user wears generallyon his/her face, a head mounted display (HMD) apparatus that is mountedon a head, an augmented reality helmet, etc. However, the disclosure isnot limited thereto, and the augmented reality device may be implementedas one of various electronic devices such as a mobile device, asmartphone, a laptop computer, a desktop computer, a tablet PC, ane-book terminal, a digital broadcasting terminal, a personal digitalassistant (PDA), a portable multimedia player (PMP), a navigationdevice, an MP3 player, a camcorder, an Internet protocol television(IPTV), a digital television (DTV), or a wearable device.

Moreover, a ‘real scene’ may be a real world scene that a user views viaan augmented reality device and may include real world object(s).

In the disclosure, a ‘virtual image’ may be an image generated throughan optical engine and may include both a still image and a video. Thevirtual image may be shown together with a real scene and may be avirtual image representing information about a real world object in thereal scene or information about an operation of an augmented realitydevice.

In the disclosure, a ‘virtual object’ refers to a partial region of avirtual image. The virtual object may indicate information related to areal world object. The virtual object may include, for example, at leastone of letters, numbers, symbols, icons, images, or animations.

The disclosure will now be described more fully with reference to theaccompanying drawings for one of ordinary skill in the art to be able toperform the disclosure without any difficulty. However, the disclosuremay be embodied in many different forms and is not limited to theembodiments of the disclosure set forth herein.

Hereinafter, embodiments of the disclosure will be described in detailwith reference to the drawings.

FIG. 1 is a conceptual view illustrating an operation by which anaugmented reality device 100 displays a blur image of a real scene,i.e., a scene, according to an embodiment of the disclosure.

Referring to FIG. 1 , the augmented reality device 100 may include acamera 110 and a glasses lens 142. The augmented reality device 100 mayobtain time-sequential image frames of the real scene by photographingimages of a real scene. In an embodiment of the disclosure, the realscene may be a work space where a user performs a work, but is notlimited to. A plurality of real world objects 1 through 9 may bearranged in the work space. The plurality of real world objects 1through 9 may include at least one preset object, i.e., at least oneobject. In an embodiment of the disclosure, the preset object may be auser-defined object. In the disclosure, the ‘user-defined object’ refersto an object pre-defined or determined as an object related to the workof the user. At least one user-defined object may be provided. Theuser-defined object may be defined by the user's selection. Examples ofthe at least one user-defined object may include, but are not limitedto, a laptop PC, a computer monitor, a keyboard, and a mouse. In anembodiment of FIG. 1 , the real world objects 1 through 9 include alaptop PC 1, computer monitors 2 and 3, a keyboard 4, and a mouse 5which may be user-defined objects pre-defined by the user. Informationabout an image of the user-defined object may be stored in a memory 130(see FIG. 2 ) of the augmented reality device 100.

The augmented reality device 100 may recognize a plurality of objectsfrom a plurality of image frames. In an embodiment of the disclosure,the augmented reality device 100 may detect the plurality of objectsfrom the plurality of image frames by using an object detection model.The object detection model may include, but is not limited to, aconvolutional neural network. The augmented reality device 100 mayidentify at least one user-defined object 11, 12, 13, 14, and 15 fromamong the plurality of objects detected from the plurality of imageframes.

The augmented reality device 100 may segment an area corresponding tothe at least one user-defined object 11, 12, 13, 14, and 15 identifiedfrom the plurality of image frames. In an embodiment of the disclosure,the augmented reality device 100 may segment the area corresponding tothe at least one user-defined object 11, 12, 13, 14, and 15 from theplurality of image frames, by using an object segmentation modelincluding a convolutional neural network model. The augmented realitydevice 100 may obtain a mask, by using an image frame in which the areacorresponding to the at least one user-defined object 11, 12, 13, 14,and 15 is removed according to a segmentation result.

The augmented reality device 100 may obtain a blur image 10, byrendering the plurality of image frames by using the mask. The blurimage 10 may be an image in which a peripheral area 20 other than thearea corresponding to the at least one user-defined object 11, 12, 13,14, and 15 is blurred.

The augmented reality device 100 may display the blur image 10 throughthe glasses lens 142. In an embodiment of the disclosure, the glasseslens 142 may be formed of a transparent material, and may be implementedas a see-through display capable of viewing the blur image 10 as well asthe plurality of real world objects 1 through 9 in the work space. In anembodiment of the disclosure, the glasses lens 142 may output the blurimage 10 projected by a display engine 144 (see FIG. 2 ) to the user,and the user may view the blur image 10 through the glasses lens 142.

When there is plurality of objects within a field of view of the user inthe work space, objects not directly related to the work from among theplurality of real world objects 1 through 9, for example, a headset 6, amobile phone 8, and a lamp 9, may attract the user's attention and mayreduce concentration. Also, when the mobile phone 8 or a tablet PC islocated in the work space, the user's attention may be attracted throughmessage notification, Internet search, Internet shopping, etc. and thework of the user may be disturbed or the user may be distracted.

Because the augmented reality device 100 according to an embodiment ofthe disclosure recognizes the at least one user-defined object 11, 12,13, 14, and 15 pre-defined as an object related to the work from theplurality of image frames obtained by photographing the work spacethrough the camera 110, and displays, to the user, the blur image 10 inwhich the peripheral area 20 other than the area corresponding to therecognized at least one user-defined object 11, 12, 13, 14, and 15 isblurred, concentration of the user on the work may be improved. Also,the augmented reality device 100 according to an embodiment of thedisclosure may provide a special visual environment and user experience(UX) related to the work through the blur image 10.

FIG. 2 is a block diagram illustrating elements of the augmented realitydevice 100, according to an embodiment of the disclosure.

Referring to FIG. 2 , the augmented reality device 100 may include thecamera 110, a processor 120, the memory 130, and a display 140. Thecamera 110, the processor 120, the memory 130, and the display 140 maybe electrically and/or physically connected to one another.

The elements illustrated in FIG. 2 are an example, and elements includedin the augmented reality device 100 are not limited to those illustratedin FIG. 2 . The augmented reality device 100 may not include some of theelements illustrated in FIG. 2 , or may further include elements notillustrated in FIG. 2 . For example, the augmented reality device 100may further include a battery for supplying driving power to the camera110, the processor 120, the memory 130, and the display 140.Alternatively, the augmented reality device 100 may further include agaze tracking sensor for obtaining data about a gaze direction of auser. Alternatively, the augmented reality device 100 may furtherinclude a hand tracking sensor for identifying the user's palm or fingerfrom an image frame and recognizing an area or a point indicated by thehand.

The camera 110 is configured to photograph a real scene, for example, awork space, and obtain an image frame. In an embodiment of thedisclosure, the camera 110 may obtain a sequence of image frames as timepasses. The camera 110 may include a lens and an image sensor. When theuser wears the augmented reality device 100, the lens may be located ina direction toward the real scene, instead of the user's face. The imagesensor may obtain a plurality of image frames by receiving lightreflected by a real world object of the real scene through the lens,converting a luminance or intensity of the received light into anelectrical signal, and imaging the electrical signal.

The camera 110 provides data of the plurality of image frames to theprocessor 120.

The processor 120 may execute one or more instructions or program codestored in the memory 130, and may perform a function and/or an operationcorresponding to the instructions or the program code. The processor 120may include a hardware component for performing arithmetic, logic, andinput/output operations and signal processing. The processor 120 mayinclude at least one of, for example, but not limited to, a centralprocessing unit, a microprocessor, a graphics processing unit, anapplication processor (AP), an application-specific integrated circuit(ASIC), a digital signal processor (DSP), a digital signal processingdevice (DSPD), a programmable logic device (PLD), or afield-programmable gate array (FPGA).

Although the processor 120 is one element in FIG. 2 , the disclosure isnot limited thereto. In an embodiment of the disclosure, one or moreprocessors instead of the processor 120 may be provided. In yet anotherembodiment, the processor 120 includes a plurality of processors.

In an embodiment of the disclosure, the processor 120 may be a dedicatedhardware chip for performing artificial intelligence (AI) learning.

Instructions and program code readable by the processor 120 may bestored in the memory 130. The memory 130 may include at least one of,for example, a flash memory type, a hard disk type, a solid state drive(SSD), a multimedia card micro type, a card-type memory (e.g., SD or XDmemory), a random-access memory (RAM), a static random-access memory(SRAM), a read-only memory (ROM), an electrically erasable programmableread-only memory (EEPROM), a programmable read-only memory (PROM), amask ROM, or a flash ROM.

Instructions or program code for performing functions or operations ofthe augmented reality device 100 may be stored in the memory 130. In anembodiment of the disclosure, at least one of instructions, analgorithm, a data structure, program code, or an application programreadable by the processor 120 may be stored in the memory 130. Theinstructions, model, algorithm, data structure, and program code storedin the memory 130 may be implemented in a programming or scriptinglanguage such as C, C++, Java, or assembler.

Instructions, an algorithm, a data structure, or program code related toan object detection model 132, an object tracking algorithm 134, anobject segmentation model 136, and an image rendering module 138 may bestored in the memory 130. A ‘model’ or ‘module’ included in the memory130 refers to a unit for processing a function or an operation performedby the processor 120, and may be implemented as software such asinstructions, an algorithm, a data structure, or program code.

In the following embodiment of the disclosure, the processor 120 may beimplemented by executing the instructions or program code stored in thememory 130.

The object detection model 132 includes instructions or program coderelated to an operation and/or a function of recognizing an object froman image frame. In an embodiment of the disclosure, the object detectionmodel 132 may be an artificial neural network model. The objectdetection model 132 may be a deep neural network model trained to detectan object through supervised learning by applying a bounding box imagedetectable as an object from tens of thousands or hundreds of millionsof input images as input data and a label value for an object detectionresult as an output ground truth. The object detection model 132 may beimplemented as, for example, region-based convolutional neural network(R-CNN), faster region-based convolutional neural network (FasterR-CNN), single shot multibox detector (SSD), YOLO v4, CenterNet, orMobileNet. However, the object detection model 132 of the disclosure isnot limited to the above deep neural network model.

The processor 120 may detect a plurality of objects from a plurality ofimage frames, by executing instructions or program code related to theobject detection model 132. In an embodiment of the disclosure, theprocessor 120 may detect the plurality of objects from a first imageframe from among the plurality of image frames by using the objectdetection model 132, and may identify at least one preset object fromamong the plurality of objects. The processor 120 may identify the atleast one preset object from among the plurality of objects detectedfrom the first image frame, based on information about a preset object.The information about the preset object may be pre-stored in the memory130. In an embodiment of the disclosure, the preset object may be auser-defined object. The ‘user-defined object’ refers to an objectpre-defined or determined as an object related to a work of a user. Theuser-defined object may be defined by the user's selection. Informationabout the user-defined object may be pre-stored in the memory 130. Theprocessor 120 may identify at least one user-defined object from amongthe plurality of objects by comparing the plurality of objects detectedfrom the first image frame with an image of the user-defined objectpre-stored in the memory 130. In an embodiment of the disclosure, theprocessor 120 may identify the at least one user-defined object from theplurality of objects by using instance recognition. A specificembodiment where the processor 120 recognizes the at least oneuser-defined object from the first image frame by using the objectdetection model 132 will be described in detail with reference to FIGS.5 and 6 .

The object tracking algorithm 134 includes instructions or program coderelated to image processing for tracking a position change in an objectrecognized from a plurality of image frames. The object trackingalgorithm 134 may track a change in an object by using featureinformation such as a size, a shape, an outline, or a color of theobject in the plurality of image frames. In an embodiment of thedisclosure, the processor 120 may track at least one object recognizedin an image frame prior to a first image frame from among the pluralityof image frames by executing instructions or program code related to theobject tracking algorithm 134, and may recognize at least oneuser-defined object from the first image frame. The processor 120 maytrack a position change in the at least one user-defined object from thefirst image frame by using the object tracking algorithm 134.

The object segmentation model 136 includes instructions or program coderelated to an operation and/or a function of segmenting an object areafrom an image. In an embodiment of the disclosure, the objectsegmentation model 136 may include a deep neural network model. In anembodiment of the disclosure, the object segmentation model 136 may beimplemented as a convolutional neural network object segmentation model.However, the disclosure is not limited thereto, and the objectsegmentation model 136 may be implemented as, for example, region-basedconvolutional neural network (R-CNN), faster region-based convolutionalneural network (Fast R-CNN), single shot multibox detector (SSD), YOLOv4, CenterNet, or MobileNet. Although the object segmentation model 136is a neural network model separate from the object detection model 132in FIG. 2 , the disclosure is not limited thereto. In another embodimentof the disclosure, the object segmentation model 136 and the objectdetection model 132 may be integrated into one neural network model.

In an embodiment of the disclosure, the processor 120 may segment anarea corresponding to at least one user-defined object from a firstimage frame, by executing instructions or program code related to theobject segmentation model 136. The processor 120 may obtain an imageframe in which the area corresponding to the at least one user-definedobject is removed according to a segmentation result. The processor 120may obtain a mask by using the obtained image frame. The ‘mask’ refersto an image for masking, modifying, or editing a specific portion of theimage frame.

In an embodiment of the disclosure, the augmented reality device 100 mayfurther include a user input interface for receiving a user inputselecting an important object from among the at least one user-definedobject. The user input interface may be, for example, a hand trackingsensor for recognizing an area or an object indicated by a user's palmor finger. In this case, the processor 120 may recognize an objectindicated by the user's finger from among the at least one user-definedobject by using the hand tracking sensor, and may select the recognizedobject as an important object. However, the disclosure is not limitedthereto, and the user input interface may include a gaze tracking sensorfor obtaining coordinate information of a gaze point at which the usergazes with both eyes, by obtaining information about a gaze direction ofthe user's both eyes. In this case, the processor 120 may recognize anobject on which the gaze point stays for a preset period of time or morefrom among the at least one user-defined object as an important object.The processor 120 may obtain a mask by segmenting the important objectselected by a user input from the first image frame.

The image rendering module 138 includes instructions or program coderelated to an operation and/or a function of generating a blur image, byrendering a plurality of image frames by using a mask. The imagerendering module 138 may blur an image by using image blurring or imagesmoothing technology. The image rendering module 138 may perform imageblurring by using at least one of, for example, average blurring,Gaussian blurring, median blurring, or bilateral filter. However, thedisclosure is not limited thereto, and the image rendering module 138may perform image blurring by using any known blurring technology.

The processor 120 may synthesize a second image frame obtained after afirst image frame with a mask by executing instructions or program coderelated to the image rendering module 138, and may obtain a blur imagein which a peripheral area other than an area corresponding to the atleast one user-defined object from among entire area of the second imageframe is blurred. In an embodiment of the disclosure, the processor 120may blur the peripheral area other than the area corresponding to the atleast one user-defined object from among all of the areas of the secondimage frame, by performing convolution of the second image frame and themask. A specific embodiment where the processor 120 segments the areacorresponding to the at least one user-defined object from the firstimage frame by using the object segmentation model 136 and obtains theblur image by using the image rendering module 138 will be described indetail with reference to FIGS. 7 and 8 .

In an embodiment of the disclosure, the augmented reality device 100 mayfurther include a user input interface for receiving a user inputdetermining a blur option including at least one of a blur degree, acolor, or a brightness about a peripheral area. The processor 120 maydetermine the blur option including at least one of the blur degree, thecolor, or the brightness based on a user input received through the userinput interface, and may blur the peripheral area from among entire areaof a second image frame based on the determined blur option.

In an embodiment of the disclosure, the processor 120 may obtain a blurimage, by performing image rendering only during a working time. Theworking time may be preset by a user. For example, the working time maybe set to 45 minutes, and a rest time may be set to 15 minutes. Theprocessor 120 may display the blur image on the display 140 only duringthe working time. A specific embodiment where the processor 120 obtainsthe blur image only during the working time and displays the blur imagewill be described in detail with reference to FIG. 9 .

In an embodiment of the disclosure, the processor 120 may obtain a blurimage, by synthesizing a peripheral area that is blurred from among allimages of a second image frame with a different color. The processor 120may synthesize a peripheral area with a different color according to awork type or a work environment. A specific embodiment where theprocessor 120 obtains the blur image by synthesizing the peripheral areawith the different color according to the type of the work and the workenvironment will be described in detail with reference to FIG. 10 .

In an embodiment of the disclosure, the processor 120 may obtain a blurimage by synthesizing a peripheral area that is blurred from amongentire area of a second image frame with a virtual image related to awork. The virtual image synthesized with the peripheral area may be avirtual object or a graphical user interface (GUI) that providesinformation related to the work. A specific embodiment where theprocessor 120 obtains the blur image by synthesizing the peripheral areawith the virtual object or the graphical UI will be described in detailwith reference to FIG. 11 .

In an embodiment of the disclosure, the augmented reality device 100 mayfurther include a user input interface for receiving a user input thatselects at least one object from among a plurality of objects recognizedfrom an image. The processor 120 may select at least one object fromamong the plurality of objects recognized from a first image frame basedon a user input received through the user input interface, and mayobtain a blur image by blurring the selected at least one object. Aspecific embodiment where the processor 120 obtains the blur image byblurring the at least one object selected by the user input will bedescribed in detail with reference to FIGS. 13 and 14 .

The processor 120 may display a blur image on the display 140 to a user.

The display 140 may include the glasses lens 142 and the display engine144.

The glasses lens 142 may be formed of a transparent material, and may beimplemented as a see-through display capable of viewing the blur imageprojected by the display engine 144 as well as a real world object in awork space. In an embodiment of the disclosure, the glasses lens 142 mayinclude a waveguide through which light of the blur image projected fromthe display engine 144 is received, light is transmitted, and an opticalpath is changed.

The display engine 144 is configured to project the blur image to thewaveguide of the glasses lens 142. The display engine 144 may perform afunction of a projector. The display engine 144 may further include anillumination optical system, an optical path converter, an image panel,a beam splitter, and a projection optical system. In an embodiment ofthe disclosure, the display engine 144 may obtain image data of the blurimage, may generate a virtual image based on the obtained image data,and may project the virtual image to the waveguide through an emissionsurface along with light output from a light source. In this case, theprocessor 120 may provide image data including RGB color and luminancevalues of a plurality of pixels constituting the virtual image to thedisplay engine 144. The display engine 144 may perform image processingby using the RGB color and luminance values of the plurality of pixels,and may project the virtual image to the waveguide by controlling thelight source.

FIG. 3 is a flowchart illustrating an operating method of the augmentedreality device 100, according to an embodiment of the disclosure.

FIG. 4 is a view illustrating an operation by which the augmentedreality device 100 obtains a blur image 420, according to an embodimentof the disclosure.

An operation by which the augmented reality device 100 obtains the blurimage 440 will be described with reference to FIGS. 3 and 4 .

Referring to FIG. 3 , in operation S310, the augmented reality device100 obtains a plurality of image frames by photographing a real scene byusing a camera. The real scene may be, for example, a work space where auser performs a work. However, the disclosure is not limited thereto. Inan embodiment of the disclosure, the augmented reality device 100 maysequentially obtain the plurality of image frames over time by using thecamera.

Referring to FIG. 4 together, the augmented reality device 100 mayobtain a first image frame 400 from among the plurality of image frames.The first image frame 400 may include a plurality of objects 401 through408.

Referring to operation S320 of FIG. 3 , the augmented reality device 100detects at least one preset object from the first image frame from amongthe plurality of image frames. Referring to FIG. 4 together, theaugmented reality device 100 may detect the plurality of objects 401through 408 from the first image frame 400 by using the object detectionmodel 132 (see FIG. 2 ). The object detection model 132 may beimplemented as, for example, region-based convolutional neural network(R-CNN), faster region-based convolutional neural network (FasterR-CNN), single shot multibox detector (SSD), YOLO v4, CenterNet, orMobileNet. However, the object detection model 132 of the disclosure isnot limited to the above deep neural network model.

The augmented reality device 100 may identify the at least one presetobject from among the plurality of objects 401 through 409 detected fromthe first image frame 400 by using the object detection model 132. Theaugmented reality device 100 may identify the at least one preset objectfrom among the plurality of objects 401 through 409 by using informationabout preset objects stored in the memory 130 (see FIG. 2 ). In anembodiment of the disclosure, the preset objects may be user-definedobjects 401, 402, 403, 404, and 405. In the disclosure, the‘user-defined objects 401, 402, 403, 404, and 405’ refer to objectspre-defined or determined as objects related to the work of the user.The user-defined objects may be defined by the user's selection.Information about the user-defined objects 401, 402, 403, 404, and 405and images of the user-defined objects 401, 402, 403, 404, and 405 maybe pre-stored in the memory 130 (see FIG. 2 ). The augmented realitydevice 100 may identify the at least one user-defined object 401, 402,403, 404, and 405 from among the plurality of objects 401 through 409,by comparing the plurality of objects 401 through 409 detected from thefirst image frame 400 with images of the user-defined objects pre-storedin the memory 130. In an embodiment of the disclosure, the augmentedreality device 100 may identify the at least one user-defined object401, 402, 403, 404, and 405 from the plurality of objects 401 through409 by using instance recognition.

Although the user-defined objects 401, 402, 403, 404, and 405 are alaptop PC 401, computer monitors 402 and 403, a keyboard 404, and amouse 405 in an embodiment of FIG. 4 , the disclosure is not limitedthereto. Also, although the number of user-defined objects 401, 402,403, 404, and 405 is 5 in FIG. 4 , this is merely an example and thedisclosure is not limited thereto. In an embodiment of the disclosure,one or more user-defined objects 401, 402, 403, 404, and 405 may beprovided.

The augmented reality device 100 may display bounding boxes B1, B2, B3,B4, and B5 surrounding the user-defined objects 401, 402, 403, 404, and405 to specify the identified user-defined objects 401, 402, 403, 404,and 405.

Referring to operation S330 of FIG. 3 , the augmented reality device 100obtains a mask, by segmenting the at least one preset object from thefirst image frame. Referring to FIG. 4 together, the augmented realitydevice 100 may specify the at least one user-defined object 401, 402,403, 404, and 405 in the bounding boxes B1, B2, B3, B4, and B5 in thefirst image frame 400 by using the object segmentation model 136 (seeFIG. 2 ). The augmented reality device 100 may identify a shape of theat least one user-defined object 401, 402, 403, 404, and 405 bydetecting an outline and a boundary line of the at least oneuser-defined object 401, 402, 403, 404, and 405, and may segment andremove, from the first image frame 400, an area corresponding to theidentified at least one user-defined object 401, 402, 403, 404, and 405.The augmented reality device 100 may obtain an image frame in which thearea corresponding to the at least one user-defined object 401, 402,403, 404, and 405 is removed from the first image frame 400 according toa segmentation result. The augmented reality device 100 may a mask 410for separating a peripheral area from the area corresponding to the atleast one user-defined object 401, 402, 403, 404, and 405 by using theobtained image frame.

Referring to operation S340 of FIG. 3 , the augmented reality device 100obtains a blur image in which the peripheral area other than the atleast one user-defined object is blurred, by rendering a second imageframe by using the mask. Referring to FIG. 4 together, the augmentedreality device 100 may synthesize the second image frame obtained afterthe first image frame 400 with the mask 410, and may obtain the blurimage 420 in which a peripheral area 422 other than the areacorresponding to the at least one user-defined object 401, 402, 403,404, and 405 from among entire area of the second image frame isblurred. In an embodiment of the disclosure, the augmented realitydevice 100 may blur the peripheral area 422 other than the areacorresponding to the at least one user-defined object from among all ofthe areas of the second image frame, by performing convolution of thesecond image frame and the mask 410.

Referring to operation S350 of FIG. 3 , the augmented reality device 100displays the blur image 4520 (see FIG. 4 ). The augmented reality device100 may project the blur image 420 to a waveguide along with lightoutput from a light source by using the display engine 144 (see FIG. 2), and may display the blur image 420 through the waveguide to the user.

FIG. 5 is a flowchart illustrating a method by which the augmentedreality device 100 identifies a user-defined object from a first imageframe, according to an embodiment of the disclosure.

Operations S510 and S520 illustrated in FIG. 5 are specified steps ofoperation S320 of FIG. 3 . Operation S510 of FIG. 5 may be performedafter operation S310 of FIG. 3 is performed. After operation S520 ofFIG. 5 is performed, operation S330 of FIG. 3 may be performed.

FIG. 6 is a view illustrating an operation by which the augmentedreality device 100 recognizes user-defined objects 601, 602, 603, 604,and 605 from a first image frame 600, according to an embodiment of thedisclosure.

An operation by which the augmented reality device 100 recognizes theuser-defined objects 601, 602, 603, 604, and 605 from the first imageframe 600 will be described with reference to FIGS. 5 and 6 together.

Referring to FIG. 5 , in operation S510, the augmented reality device100 detects a plurality of objects from a first image frame by using anobject detection model. Referring to FIG. 6 together, the augmentedreality device 100 may detect bounding boxes B1 through B13 in which theplurality of objects are detected from the first image frame 600, byperforming inference of inputting the first image frame 600 from among aplurality of image frames obtained through the camera 110 (see FIG. 2 )to the object detection model 132 (see FIG. 2 ). In an embodiment of thedisclosure, the object detection model 132 may be a deep neural networkmodel trained to detect an object through supervised learning byapplying a bounding box image detectable as an object from tens ofthousands or hundreds of millions of input images as input data andapplying a label value for an object detection result as an outputground truth. The object detection model 132 may be implemented as, forexample, region-based convolutional neural network (R-CNN), fasterregion-based convolutional neural network (Faster R-CNN), single shotmultibox detector (SSD), YOLO v4, CenterNet, or MobileNet. However, theobject detection model 132 of the disclosure is not limited to the abovedeep neural network model.

The augmented reality device 100 may input the first image frame 600 tothe object detection model 132 including a pre-trained deep neuralnetwork model, and may detect the plurality of bounding boxes B1 throughB13 including the plurality of objects according to an inference resultthrough the object detection model 132. In an embodiment of thedisclosure, the augmented reality device 100 may recognize the pluralityof bounding boxes B1 through B13 by using a detection result from animage frame obtained prior to the first image frame 600 by using theobject tracking algorithm 134 (see FIG. 2 ). The object trackingalgorithm 134 may track a change in an object by using featureinformation such as a size, a shape, an outline, or a color of theobject in a plurality of image frames. In an embodiment of thedisclosure, the augmented reality device 100 may track at least oneobject recognized in the image frame prior to the first image frame 600from among the plurality of image frames by using the object trackingalgorithm 134, and may recognize the plurality of bounding boxes B1through B13 from the first image frame 600.

Referring to operation S520 of FIG. 5 , the augmented reality device 100identifies at least one user-defined object pre-defined as an objectrelated to a work of a user from among the plurality of objects.Referring to FIG. 6 together, the augmented reality device 100 mayidentify the bounding boxes B1, B2, B3, B4, and B5 including the atleast one user-defined object 601, 602, 603, 604, and 605, by comparingthe plurality of objects included in the recognized plurality ofbounding boxes B1 through B13 with images of user-defined objectspre-stored in the memory 130 (see FIG. 2 ). In an embodiment of thedisclosure, the augmented reality device 100 may identify the boundingboxes B1, B2, B3, B4, and B5 including the at least one user-definedobject 601, 602, 603, 604, and 605 from the plurality of bounding boxesB1 through B13 by using instance recognition. In the disclosure, the‘user-defined objects 601, 602, 603, 604, and 605’ refer to objectspre-defined or determined as objects related to the work of the user.

FIG. 7 is a view illustrating an operation by which the augmentedreality device 100 recognizes user-defined objects 701, 702, 703, 704,and 705 from a first image frame 700, according to an embodiment of thedisclosure.

Referring to FIG. 7 , the augmented reality device 100 may accuratelyspecify the user-defined objects 701, 702, 703, 704, and 705 in boundingboxes B1, B2, B3, B4, and B5 recognized from the first image frame 700by using the object segmentation model 136 (see FIG. 2 ). The objectsegmentation model 136 may include a deep neural network model forsegmenting an object from an image. The object segmentation model 136may be implemented as, for example, but not limited to, a convolutionalneural network object segmentation model. Alternatively, the objectsegmentation model 136 may be implemented as region-based convolutionalneural network (R-CNN), faster region-based convolutional neural network(Faster R-CNN), single shot multibox detector (SSD), YOLO v4, CenterNet,or MobileNet.

The augmented reality device 100 may identify areas of the user-definedobjects 701, 702, 703, 704, and 705 by detecting outlines and boundarylines of the user-defined objects 701, 702, 703, 704, and 705 in thebounding boxes B1, B2, B3, B4, and B5 detected from the first imageframe 700 by using the object segmentation model 136. The augmentedreality device 100 may segment the areas of the user-defined objects701, 702, 703, 704, and 705 from the first image frame 700.

FIG. 8 is a view illustrating an operation by which the augmentedreality device 100 obtains a blur image 820, according to an embodimentof the disclosure.

Referring to FIG. 8 , the augmented reality device 100 may segment andremove an area corresponding to at least one user-defined object 801,802, 803, 804, and 805 from a first image frame 800 by using the objectsegmentation model 136 (see FIG. 2 ). The augmented reality device 100may obtain an image frame in which the area corresponding to the atleast one user-defined object 801, 802, 803, 804, and 805 is removedfrom the first image frame 800 according to a segmentation result. Theaugmented reality device 100 may obtain a mask 810 for separating thearea corresponding to the at least one user-defined object 801, 802,803, 804, and 805 from a peripheral area by using the obtained imageframe.

The mask 810 may be an image for masking, modifying, or editing aspecific portion of the image frame. Referring to an embodiment of FIG.8 , in the mask 810, the area corresponding to the at least oneuser-defined object 801, 802, 803, 804, and 805 may be processed to betransparent, and the peripheral area other than the at least oneuser-defined object 801, 802, 803, 804, and 805 may be processed to beblack.

The augmented reality device 100 may obtain a blur image 820, byrendering a plurality of image frames by using the mask 810. In anembodiment of the disclosure, the augmented reality device 100 maysynthesize a second image frame obtained after the first image frame 800with the mask 810, and may obtain the blur image 820 in which aperipheral area 822 other than the area corresponding to the at leastone user-defined object 801, 802, 803, 804, and 805 from among entirearea of the second image frame is blurred. In an embodiment of thedisclosure, the augmented reality device 100 may blur the peripheralarea 822 other than the area corresponding to the at least oneuser-defined object 801, 802, 803, 804, and 805 from among all of theareas of the second image frame, by performing convolution of the secondimage frame and the mask 810.

In an embodiment of the disclosure, the augmented reality device 100 mayblur the plurality of image frames through rendering using imageblurring or image smoothing technology. The augmented reality device 100may perform image blurring by using at least one of, for example,average blurring, Gaussian blurring, median blurring, or bilateralfilter.

FIG. 9 is a view for describing an operation by which the augmentedreality device 100 displays different images as time passes, accordingto an embodiment of the disclosure.

Referring to FIG. 9 , the augmented reality device 100 may displaydifferent images over time through the glasses lens 142. In anembodiment of the disclosure, the augmented reality device 100 maydisplay a blur image 900 in which a peripheral area other than at leastone user-defined object is blurred, by performing image rendering onlyduring a working time. The ‘working time’ refers to a time preset for auser to perform a work. The working time may be determined by a userinput. The working time may be determined to be, for example, but notlimited to, 45 minutes.

Referring to an embodiment of FIG. 9 , the processor 120 (see FIG. 2 )of the augmented reality device 100 may obtain the blur image 900 inwhich the peripheral area other than an area corresponding to the atleast one user-defined object is blurred by rendering an image frameduring a working time between a start time t₀ and a first time t₁, andmay display the blur image 900 through the glasses lens 142. In anembodiment of the disclosure, the processor 120 may display the blurimage 900 by synthesizing the blurred peripheral area with a specificcolor. The synthesized color may be determined by a user input. Forexample, the processor 120 may display the blur image 900, bysynthesizing a green color with the blurred peripheral area.

The processor 120 may display an image 910 of a work space, withoutperforming rendering such as blurring on the image frame during a resttime between the first time t₁ and a second time t₂. In an embodiment ofthe disclosure, the processor 120 may display the image 910 of the workspace obtained by the camera 110 (see FIG. 2 ) in its original statewithout additional rendering. The rest time between the first time t₁and the second time t₂ may be, for example, but is not limited to, 15minutes.

The processor 120 may blur the peripheral area other than the at leastone user-defined object by rendering the image frame during a workingtime between the second time t₂ and a third time t₃, and may display theblur image 900. Like in the working time between the start time t₀ andthe first time t₁, the processor 120 may obtain the blur image 900 bysynthesizing the blurred peripheral area even during the working timebetween the second time t₂ and the third time t₃ with a color determinedby a user input, and may display the blur image 900 through the glasseslens 142.

The processor 120 may display the image 910 of the work space, withoutperforming rendering such as blurring on the image frame during a resttime between the third time t₃ and a fourth time t₄. The processor 120may display the blur image 900, by rendering the image frame during aworking time between the fourth time t₄ and a fifth time t₅.

Because the augmented reality device 100 according to an embodiment ofFIG. 9 selectively displays the blur image 900 in which the peripheralarea other than the area corresponding to the at least one user-definedobject is blurred only during the working time set by the user, atechnical effect of improving concentration on the work of the userduring the working time is provided. Also, because the augmented realitydevice 100 according to an embodiment of the disclosure synthesizes anddisplays the blurred peripheral area with a specific color, a specialvisual environment and special user experience (UX) related to the workmay be provided to the user.

FIG. 10 is a view for describing an operation by which the augmentedreality device 100 displays a different blur image according to a worktype, according to an embodiment of the disclosure. The different worktypes may require or emphasize the use of different objects. Forexample, one work type, WORK 1, may require or emphasize the use of justtwo monitors, a keyboard, and a mouse. WORK 2 is another work type thatmay require or emphasize the use of one monitor, a keyboard, a mouse,and a laptop pc. Another work type, WORK 3, may require or emphasize theuse of only the laptop pc.

Referring to FIG. 10 , the augmented reality device 100 may display blurimages 1010, 1020, and 1030 synthesized with different colors throughthe glasses lens 142. The processor 120 (see FIG. 2 ) of the augmentedreality device 100 may obtain the blur images 1010, 1020, and 1030 bysynthesizing a peripheral area that is blurred from among entire area ofan image frame with a different color, and may display the obtained blurimages 1010, 1020, and 1030 through the glasses lens 142.

In an embodiment of the disclosure, the processor 120 may synthesize theblurred peripheral area with a different color according to a work typeor a work environment. In an embodiment of FIG. 10 , the processor 120may display the first blur image 1010 generated by synthesizing theperipheral area other than an area corresponding to at least oneuser-defined object from among all of the areas of the image with agreen color during a time from a start time t₀ to a first time t₁ toperform a first work. The processor 120 may display the second blurimage 1020 generated by synthesizing the peripheral area other than thearea corresponding to the at least one user-defined object from amongall of the areas of the image frame with a blue color during a time froma second time t₂ to a third time t₃ to perform a second work, and maydisplay the third blur image 1030 generated by synthesizing theperipheral area with a red color during a time from the third time t₃ toa fourth time t₄ to perform a third work. The processor 120 may displaythe first blur image 1010 during a time from the fourth time t₄ to afifth time t₅ to perform the first work.

In an embodiment of the disclosure, the processor 120 may determine theblurred peripheral area based on a work type and a work environment. Forexample, for the first blur image 1010 displayed while the first work isperformed, a peripheral area other than a computer monitor from amongall of the areas of the image frame may be blurred; for the second blurimage 1020 displayed while the second work is performed, a peripheralarea other than a laptop PC and one computer monitor from among all ofthe areas of the image frame may be blurred; and for the third blurimage 1030 displayed while the third work is performed, a peripheralarea other than the laptop PC from among all of the areas of the imageframe may be blurred.

The processor 120 may display an image 1000 of a work state in anoriginal state, without performing image rendering during a rest time,not a working time (CHATTING 1).

A user may not perform only one work but may perform multiple works. Themultiple works may be different in types, work environments, and toolsused for the works (e.g., a laptop PC, a desktop PC, and a tablet PC).Because the augmented reality device 100 according to an embodiment ofFIG. 10 displays the blur images 1010, 1020, and 1030 in which aperipheral area is synthesized with a different color according to atype of each of a plurality of works and work environments, when theuser performs various works, a technical effect of facilitatingswitching between works and improving concentration on a switched workis provided. Also, because the augmented reality device 100 according toan embodiment of the disclosure displays the blur images 1010, 1020, and1030 of different colors according to types of works or workenvironments, a special visual environment for a work may be provided.

FIG. 11 is a view illustrating an operation by which the augmentedreality device 100 displays a virtual object or a graphical UI in ablurred peripheral area, according to an embodiment of the disclosure.

Referring to FIG. 11 , the augmented reality device 100 may display ablur image 1100 obtained by synthesizing a virtual image 1120 with aperipheral area 1110 other than an area corresponding to at least oneuser-defined object 1101, 1102, 1103, 1104, and 1105 from among entirearea of an image frame. In an embodiment of the disclosure, theprocessor 120 (see FIG. 2 ) of the augmented reality device 100 mayperform rendering for synthesizing the virtual image 1120 with theperipheral area 1110 so that the virtual image 1120 is overlaid on theperipheral area 1110 from among entire area of the blur image 1100. Inan embodiment of the disclosure, the processor 120 may determine aposition of the virtual image 1120 so that the virtual image 1120 doesnot overlap the area of the at least one user-defined object 1101, 1102,1103, 1104, and 1105.

The virtual image 1120 may be a virtual object or a graphical UI thatprovides information related to a work. For example, a first virtualimage 1121 may be a widget that records text or coding instructionsrelated to the work, a second virtual image 1122 may be a calendar UI, athird virtual image 1123 may be a memo UI, and a fourth virtual image1124 may be a timer UI. The virtual image 1120 illustrated in FIG. 11 ismerely an example, and is not limited to the widget, the calendar UI,the memo UI, and the timer UI illustrated in FIG. 11 .

Because the augmented reality device 100 according to an embodiment ofFIG. 11 blurs the peripheral area 1110 other than the at least oneuser-defined object 1101, 1102, 1103, 1104, and 1105, and displays thevirtual image 1120 that provides information related to the work in theperipheral area 1110 that is blurred, a technical effect of improving auser's work efficiency is provided.

FIG. 12 is a flowchart illustrating a method by which the augmentedreality device 100 obtains a blur image according to a blur optiondetermined based on a user input, according to an embodiment of thedisclosure.

Operations S1210 and S1220 illustrated in FIG. 12 are specified steps ofoperation S340 of FIG. 3 . Operation S1210 of FIG. 12 may be performedafter operation S330 of FIG. 3 . After operation S1220 of 12 isperformed, operation S350 of FIG. 3 may be performed.

In operation S1210, the augmented reality device 100 determines a bluroption including at least one of a blur degree, a color, or a brightnessbased on a user input. In an embodiment of the disclosure, the augmentedreality device 100 may display a virtual image including a graphical UIfor selecting the blur option, and may receive a user input that selectsthe blur option including at least one of the blur degree, the color, orthe brightness through the virtual image. In an embodiment of thedisclosure, the augmented reality device 100 may include a hand trackingsensor for recognizing a position or an area indicated by a user'sfinger. In this case, the augmented reality device 100 may receive auser input that selects the blur option, by recognizing a position ofthe finger indicating the graphical UI by using the hand trackingsensor. The augmented reality device 100 may determine the blur optionincluding at least one of the blur degree, the color, or the brightnessrelated to a peripheral area other than at least one user-defined objectfrom among entire area of a blur mage, based on a user input.

In operation S1220, the augmented reality device 100 performs imagerendering for blurring the peripheral area based on the determined bluroption. The augmented reality device 100 may obtain the blur image byperforming image rendering on the peripheral area based on at least oneof the blur degree, the color, or the brightness related to theperipheral area determined by the user input.

FIG. 13 is a flowchart illustrating a method by which the augmentedreality device 100 obtains a blur image by blurring an object selectedby a user input, according to an embodiment of the disclosure.

FIG. 14 is a view illustrating an operation by which the augmentedreality device 100 obtains a blur image 1410 by blurring an objectselected by a user input, according to an embodiment of the disclosure.

An operation of the augmented reality device 100 will be described withreference to FIGS. 13 and 14 together.

Referring to operation S1310 of FIG. 13 , the augmented reality device100 obtains a plurality of image frames by photographing a real scene byusing the camera 110 (see FIG. 2 ). The real scene may be, for example,but is not limited to, a work space where a user performs a work.Operation S1310 is the same as operation S310 of FIG. 3 , and thus, arepeated description will be omitted.

In operation S1320, the augmented reality device 100 detects a pluralityof objects from a first image frame from among the plurality of imageframes. In an embodiment of the disclosure, the processor 120 (see FIG.2 ) of the augmented reality device 100 may detect the plurality ofobjects from the first image frame from among the plurality of imageframes by using the object detection model 132 (see FIG. 2 ). The objectdetection model 132 may be implemented as, for example, region-basedconvolutional neural network (R-CNN), faster region-based convolutionalneural network (Faster R-CNN), single shot multibox detector (SSD), YOLOv4, CenterNet, or MobileNet. However, the object detection model 132 ofthe disclosure is not limited to the above deep neural network model.Referring to FIG. 14 together, the processor 120 may recognize aplurality of objects 1401, 1402, 1403, 1404, 1405, and 1406 from a firstimage frame 1400 by using the object detection model 132.

In operation S1330, the augmented reality device 100 selects at leastone object from among the plurality of objects recognized from the firstimage frame based on a user input. In an embodiment of the disclosure,the augmented reality device 100 may further include a user inputinterface for receiving a user input that selects at least one objectfrom among the plurality of objects recognized from the first imageframe. The user input interface may be a hand tracking sensor forrecognizing an area or an object indicated by the user's palm or finger.In this case, the processor 120 of the augmented reality device 100 mayrecognize at least one object indicated by the user's finger from amongthe plurality of objects by using the hand tracking sensor, and mayselect the recognized at least one object.

Referring to FIG. 14 together, the processor 120 may recognize theuser's finger indicating the first object 1401 and the sixth object 1406from among the plurality of objects 1401, 1402, 1403, 1404, and 1405recognized from the first image frame 1400 through the hand trackingsensor. The processor 120 may select the first object 1401 and the sixthobject 1406 recognized by the hand tracking sensor. In an embodiment ofthe disclosure, the processor 120 may display bounding boxes B1 and B2including the first object 1401 and the sixth object 1406 that areselected on the first image frame 1400.

However, the disclosure is not limited thereto, and the augmentedreality device 100 may include a gaze tracking sensor for obtainingcoordinate information of a gaze point at which the user gazes with botheyes, by obtaining information about a gaze direction of the user's botheyes. In this case, the processor 120 may recognize an object on whichthe gaze point stays for a preset period of time or more from among theplurality of objects 1401, 1402, 1403, 1404, and 1405, and may selectthe recognized object.

In operation S1340, the augmented reality device 100 obtains a blurimage, by performing rendering for blurring the at least one objectselected by the user input. Referring to FIG. 14 together, the processor120 of the augmented reality device 100 may obtain the blur image 1410by segmenting the first object 1401 and the sixth object 1406 selectedby the user input from the first image frame 1400 by using the objectsegmentation model 136 (see FIG. 2 ) and then blurring image areas 1420and 1422 corresponding to the first object 1401 and the sixth object1406. In the blur image 1410, only the first area 1420 corresponding tothe first object 1401 and the second area 1422 corresponding to thesixth object 1406 may be blurred, and the remaining objects 1402, 1403,1404, and 1405 and a peripheral area may not be blurred.

In operation S1350, the augmented reality device 100 displays the blurimage 1410 (see FIG. 14 ).

Because the augmented reality device 100 according to an embodiment ofFIGS. 13 and 14 directly selects an object that may disturb the work ofthe user or reduce concentration, for example, a mobile phone (the sixthobject 1406 in FIG. 14 ), and blurs the selected object, the blur image1410 customized by the user may be obtained and displayed. Accordingly,the augmented reality device 100 according to an embodiment of thedisclosure may improve concentration on the work of the user and mayprovide a user experience related to a customized work environment.

A program executed by the augmented reality device 100 described in thedisclosure may be implemented in hardware, software, and/or acombination of hardware and software. The program may be executed by anysystem capable of executing computer-readable instructions.

The software may include a computer program, code, instructions, or acombination of one or more thereof, and may configure a processingdevice to operate as desired or instruct the processing deviceindependently or collectively.

The software may be implemented as a computer program includinginstructions stored in a computer-readable storage medium. Examples ofthe computer-readable storage medium include a magnetic storage medium(e.g., a read-only memory (ROM), a random-access memory (RAM), a floppydisk, or a hard disk), and an optical recording medium (e.g., a compactdisc ROM (CD-ROM), or a digital versatile disc (DVD)). Thecomputer-readable storage medium may be distributed in computer systemsconnected in a network so that computer-readable code is stored andexecuted in a distributed fashion. The medium may be computer-readable,may be stored in a memory, and may be executed by a processor.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. In this case, “non-transitory” means thatthe storage medium does not include a signal and is tangible but doesnot distinguish whether data is semi-permanently or temporarily storedin the storage medium. For example, the ‘non-transitory storage medium’may include a buffer in which data is temporarily stored.

Also, a program according to embodiments of the disclosure may beprovided in a computer program product. The computer program product isa product purchasable between a seller and a purchaser.

The computer program product may include a software program and acomputer-readable storage medium in which the software program isstored. For example, the computer program product may include a softwareprogram-type product (e.g., a downloadable application) electronicallydistributed by a manufacturer of the augmented reality device 100 or anelectronic market (e.g., Samsung Galaxy Store or Google Play store). Forelectronic distribution, at least a portion of the software program maybe stored in a storage medium or temporarily generated. In this case,the storage medium may be a server of the manufacturer of the augmentedreality device 100, a server of the electronic market, or a storagemedium of a relay server temporarily storing the software program.

The computer program product may include a storage medium of a server ora storage medium of the augmented reality device 100 in a systemincluding the server and/or the augmented reality device 100.Alternatively, when there is a third device (e.g., a mobile device or awearable device) communicatively connected to the augmented realitydevice 100, the computer program product may include a storage medium ofthe third device. Alternatively, the computer program product mayinclude a software program transmitted from the augmented reality device100 to the server or the third device, or transmitted from the thirddevice to the augmented reality device 100.

In this case, at least one of the augmented reality device 100, theserver, or the third device may execute the computer program product toperform a method according to embodiments of the disclosure.Alternatively, any one of the augmented reality device 100, the server,and the third device may execute the computer program product to performthe method according to the embodiments of the disclosure in adistributed fashion.

For example, the augmented reality device 100 may execute the computerprogram product stored in the memory 130 (see FIG. 2 ) to controlanother electronic device (e.g., a mobile device or a wearable device)communicatively connected to the augmented reality device 100 to performthe method according to the embodiments of the disclosure.

In another example, the third device may execute the computer programproduct to control an electronic device communicatively connected to thethird device to perform the method according to the embodiments of thedisclosure.

In the case that the third device executes the computer program product,the third device may download the computer program product from theaugmented reality device 100 and may execute the downloaded computerprogram product. Alternatively, the third device may execute thecomputer program product provided in a pre-loaded state to perform themethod according to the embodiments of the disclosure.

Although the embodiments have been described by the limited embodimentsand the drawings as described above, various modifications andvariations are possible by one of ordinary skill in the art from theabove description. For example, the described techniques may beperformed in a different order from the described method, and/or thedescribed elements such as a computer system and a module may becombined or integrated in a different form from the described method, ormay be replaced or substituted by other elements or equivalents toachieve appropriate results.

1. An augmented reality device comprising: a camera configured to obtaina plurality of image frames of a scene; a display; a memory storinginformation about a preset object; and at least one processor, whereinthe at least one processor is configured to recognize at least onepreset object corresponding to the information about the preset objectfrom a first image frame from among the plurality of image framesobtained through the camera, obtain a mask by segmenting an area of therecognized at least one preset object from the first image frame, obtaina blur image in which an area other than the area of the at least onepreset object is blurred, the blur image being obtained by renderingusing the mask, a second image frame obtained after the first imageframe, and control the display to display the blur image.
 2. Theaugmented reality device of claim 1, wherein the at least one processoris further configured to: detect a plurality of objects from the firstimage frame by using an object detection model, and identify the atleast one preset object from among the plurality of objects, wherein theat least one preset object is a user-defined object pre-defined as anobject related to a work of a user.
 3. The augmented reality device ofclaim 1, further comprising a user input interface configured to receivea user input that selects an object from among the recognized at leastone preset object, wherein the at least one processor is furtherconfigured to obtain the mask by segmenting, from the first image frame,an area corresponding to the object selected based on the user inputreceived through the user input interface.
 4. The augmented realitydevice of claim 1, wherein the at least one processor is furtherconfigured to: obtain the blur image, by performing image rendering onlyduring a preset working time, and display the blur image on the displayonly during the preset working time.
 5. The augmented reality device ofclaim 1, wherein the at least one processor is further configured tosynthesize the blurred area with a color according to a work type or awork environment.
 6. The augmented reality device of claim 1, whereinthe at least one processor is further configured to synthesize theblurred area with a virtual object or a graphical user interface (GUI)that provides information related to a work.
 7. The augmented realitydevice of claim 1, further comprising a user input interface configuredto receive a user input that selects at least one object from among aplurality of objects recognized from the first image frame, wherein theat least one processor is further configured to perform image renderingfor blurring the at least one object selected by the user input receivedthrough the user input interface.
 8. An operating method of an augmentedreality device, the operating method comprising: obtaining a pluralityof image frames by photographing a scene by using a camera; recognizingat least one preset object corresponding to information about a presetobject from a first image frame from among the plurality of imageframes; obtaining a mask, by segmenting an area of the recognized atleast one preset object from the first image frame; obtaining a blurimage in which an area other than the area of the at least one presetobject is blurred, the blur image being obtained by rendering using themask, a second image frame obtained after the first image frame; anddisplaying the blur image.
 9. The operating method of claim 8, whereinthe recognizing of the at least one preset object comprises: detecting aplurality of objects from the first image frame by using an objectdetection model; and identifying the at least one preset object fromamong the plurality of objects, wherein the at least one preset objectis a user-defined object pre-defined as an object related to a work of auser.
 10. The operating method of claim 8, wherein the obtaining of themask comprises: selecting an object from among the recognized at leastone preset object, based on a user input; and obtaining the mask, bysegmenting an area corresponding to the selected area from the firstimage frame.
 11. The operating method of claim 8, wherein the obtainingof the blur image comprises obtaining the blur image, by performingimage rendering only during a preset working time, and the displaying ofthe blur image comprises displaying the blur image only during thepreset working time.
 12. The operating method of claim 8, wherein theobtaining of the blur image comprises synthesizing the blurred area witha color according to a work type or a work environment.
 13. Theoperating method of claim 8, wherein the obtaining of the blur imagecomprises synthesizing the blurred area with a virtual object or agraphical user interface (GUI) that provides information related to awork.
 14. The operating method of claim 8, further comprising selectingat least one object from among a plurality of objects recognized fromthe first image frame based on a user input, wherein the obtaining ofthe blur image comprises performing image rendering for blurring the atleast one object selected by the user input.
 15. A computer programproduct comprising a computer-readable storage medium, wherein thecomputer-readable storage medium comprises instructions readable by anaugmented reality device to cause the augmented reality device to:obtain a plurality of image frames by photographing a scene by using acamera; recognize at least one preset object corresponding toinformation about a preset object from a first image frame from amongthe plurality of image frames; obtain a mask, by segmenting an area ofthe recognized at least one preset object from the first image frame;obtain a blur image in which an area other than the area of the at leastone preset object is blurred, the blur image being obtained byrenderingy using the mask, a second image frame obtained after the firstimage frame; and display the blur image.